Summary

PyExecutor reads TLLM_TORCH_PROFILE_TRACE directly and every rank calls torch_profiler.export_chrome_trace() on the same path. At TP/PP/DP > 1 the concurrent writes interleave and the resulting file fails to parse in Chrome tracing / Perfetto (bad control character / unterminated string at the byte where one rank's output overran another's).

Fix: append the rank to the env-provided path before the first use so each rank writes to its own file.

# tensorrt_llm/_torch/pyexecutor/py_executor.py:886
torch_trace_path = os.environ.get(PROFILE_TRACE_ENV_VAR_NAME, None)
if torch_trace_path is not None:
    trace_base, trace_ext = os.path.splitext(torch_trace_path)
    torch_trace_path = f"{trace_base}-rank-{self.global_rank}{trace_ext}"

TLLM_TORCH_PROFILE_TRACE=/tmp/trace.json now produces /tmp/trace-rank-0.json, /tmp/trace-rank-1.json, etc. — same convention SGLang's scheduler_profiler_mixin already uses (user supplies a base, runtime adds the per-rank suffix automatically).

Why this matters

• The existing behavior is unusable at any TP > 1 — the trace file is silently corrupted, not detected at write time.
• The published docs at https://nvidia.github.io/TensorRT-LLM/performance/perf-analysis.html make no mention of TP > 1 caveats — users following the docs verbatim hit silent corruption.
• Same bug, same fix concept was opened as Fix: Prevent trace file overwrite across ranks in torch profiler when using multiple GPUs #9022 in Nov 2025 (clintg6:fix/multi-gpu-torch-profiling). It received only a coderabbitai bot review and was closed by the author 6 days later without human engagement. This PR re-opens with a smaller diff (8 lines vs ~15+ refactor) and fresh validation evidence.

Validation

Reproduced on TRT-LLM 1.3.0rc11 with TP=8 on 8×H200 serving zai-org/GLM-5.1-FP8:

Before patch (single shared path):

$ ls -la /tmp/trtllm-trace.json
-rw-r--r-- 1 dynamo root 26635069 Apr 23 09:22 trtllm-trace.json
$ python3 -c "import json; json.load(open('/tmp/trtllm-trace.json'))"
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 42862 column 6 (char 2452074)

The corrupt byte range contains "name": "void at::native::vectorized_elem truncated mid-string by another rank's { for the next event.

After patch (per-rank paths, same env value, same workload):

$ ls -la /tmp/trtllm-trace-rank-*.json
-rw-r--r-- 1 dynamo root 26635133 Apr 24 02:57 /tmp/trtllm-trace-rank-0.json
-rw-r--r-- 1 dynamo root 26633450 Apr 24 02:57 /tmp/trtllm-trace-rank-1.json
-rw-r--r-- 1 dynamo root 26634910 Apr 24 02:57 /tmp/trtllm-trace-rank-2.json
-rw-r--r-- 1 dynamo root 26634923 Apr 24 02:57 /tmp/trtllm-trace-rank-3.json
-rw-r--r-- 1 dynamo root 26633834 Apr 24 02:57 /tmp/trtllm-trace-rank-4.json
-rw-r--r-- 1 dynamo root 26633499 Apr 24 02:57 /tmp/trtllm-trace-rank-5.json
-rw-r--r-- 1 dynamo root 26634943 Apr 24 02:57 /tmp/trtllm-trace-rank-6.json
-rw-r--r-- 1 dynamo root 26633389 Apr 24 02:57 /tmp/trtllm-trace-rank-7.json

$ python3 -c "import json; print(len(json.load(open('/tmp/trtllm-trace-rank-0.json'))['traceEvents']))"
63079

Distinct sizes confirm no shared clobbering; rank-0 parses with 63,079 events.

Backwards compatibility

• TLLM_TORCH_PROFILE_TRACE env name unchanged.
• TP=1 behavior changes: file is now <base>-rank-0<ext> instead of <base>. This is the same compromise SGLang made and is the only sane disambiguation if you ever scale to multi-rank.

Test plan

• Smoke: TP=8, 500-token decode, 8 distinct files, all parse with json.load
• No source patch beyond the 8-line block at py_executor.py:886
• No env-var change required by callers
• CI

Reviewers

cc @NVIDIA/trt-llm-torch-runtime-devs @byshiue @xxi-nv — re-opening the multi-rank torch-profiler trace fix from #9022 (which went stale without human review) with a smaller diff and concrete reproducer. Would appreciate eyes here so distributed profiling stops silently corrupting traces.

Summary by CodeRabbit

• Bug Fixes
  • Fixed torch profiler trace export to generate rank-specific filenames in multi-rank environments, preventing file corruption and ensuring Chrome tracing/Perfetto parsing works correctly when tracing is enabled.